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In recent years there has been increasing interest in the phe- 
nomena of “gaming the system,” where a learner attempts to 
succeed in an educational environment by exploiting proper- 
ties of the system’s help and feedback rather than by attempt- 
ing to learn the material. Developing environments that 
respond constructively and effectively to gaming depends 
upon understanding why students choose to game. In this arti- 
cle, we present three studies, conducted with two different 
learning environments, which present evidence on which stu- 
dent behaviors, motivations, and emotions are associated with 
the choice to game the system. We also present a fourth study 
to determine how teachers’ perspectives on gaming behavior 
are similar to, and different from, researchers’ perspectives 
and the data from our studies. We discuss what motivational 
and attitudinal patterns are associated with gaming behavior 
across studies, and what the implications are for the design of 
interactive learning environment. 
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In recent years, there has been increasing evidence that students choose to 
use interactive learning environments in a surprising variety of ways (Wood & 
Wood, 1999; Aleven, McLaren, Roll, & Koedinger, 2004; Baker, Corbett, 
Koedinger, Wagner, et al., 2004; Mostow et al., 2002; Arroyo & Woolf, 2005; 
Stevens & Soller, 2005), and that some choices are associated with poorer 
learning (Baker, Corbett, Koedinger, Wagner, et al., 2004; Beck, 2005; 
Aleven, McLaren, Roll, & Koedinger, 2006). In particular, one category of 
behavior, termed “gaming the system,” has been repeatedly found to be asso- 
ciated with poorer learning (Baker, Corbett, Koedinger, Wagner, et al., 2004; 
Baker, Roll, Corbett, & Koedinger, 2005; Beck; Walonoski & Heffernan, 
2006a). Baker (2005) defined gaming the system as “attempting to succeed in 
an educational environment by exploiting properties of the system rather than 
by learning the material and trying to use that knowledge to answer correctly.” 

Gaming behaviors have been observed in a variety of types of learning 
environments, from educational games (Klawe, 1998; Magnussen & Mis- 
feldt, 2004) to online course discussion forums (Cheng & Vassileva, 2005), 
and have been repeatedly documented in one type of interactive learning 
environment, intelligent tutoring systems (Aleven, 2001; Baker, Corbett, 
Koedinger, & Wagner, et al., 2004; Beck, 2005; Mostow et al., 2002; Murray 
& vanLehn, 2005; Schofield, 1995; Wood & Wood, 1999). Across the sys- 
tems studied, a reasonably substantial minority of students (10-40%) appear 
to engage in some variety of gaming behavior, at least some of the time. 

Within Cognitive Tutors and ASSISTments, the two types of intelligent 
tutoring systems we investigate in this article, gaming the system consists of 
the following behaviors: 

1. quickly and repeatedly asking for help until the tutor gives the stu- 
dent the correct answer (Wood & Wood, 1999; Aleven, 2001); and 


2. inputting a sequence of answer attempts quickly and systematical- 
ly. For instance, systematically guessing numbers in order 
(1,2,3,4...) or clicking every checkbox within a set of multiple- 
choice answers, until the tutor identifies a correct answer and 
allows the student to advance. 


Other examples of gaming the system include choosing to work on mate- 
rial which the student has already memorized (personal communication, 
Jack Mostow), restarting problems when a tutoring system only saves at the 
end of each problem, and intentionally posting irrelevant material to online 
course discussion (Cheng & Vassileva, 2005). 

Recent work to model this behavior has resulted in systems, which can 
effectively detect gaming behaviors in a variety of intelligent tutoring systems 
(Baker, Corbett, & Koedinger, 2004; Beck, 2005; Baker, Corbett, Koedinger, 
& Roll, 2006; Walonoski & Heffernan, 2006a; Johns & Woolf, 2006; Beal, 
Qu, & Lee, 2006), using a considerable variety of modeling frameworks and 
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approaches. The ability to effectively detect gaming has spurred the develop- 
ment of systems that attempt to respond to gaming, when it occurs. Prior to 
this work, most attempts to remediate gaming consisted of attempting to make 
it impossible for students to game (Aleven, 2001; Murray & vanLehn, 2005), 
but it was found that this approach both reduced the usefulness of help features 
for nongaming students (Baker, Corbett, Koedinger, & Wagner, 2004) and that 
gaming students discovered new strategies for gaming that worked despite the 
system redesigns (Murray & vanLehn, 2005). Newer systems have responded 
to gaming using models that identified exactly when and which students game 
the system, and by producing visualizations of student gaming that were visi- 
ble to both the student and their teacher, instant feedback messages that noti- 
fy the student that they are gaming, and/or supplementary exercises on exact- 
ly the material the students have bypassed through gaming. These systems 
have been successful in reducing the prevalence of gaming behavior 
(Walonoski & Heffernan, 2006b; Baker, Corbett, Koedinger, Evenson, et al., 
2006), and in improving gaming students’ learning (Baker, Corbett, 
Koedinger, Evenson, et al., 2006), over short (approximately week-long) sec- 
tions of intelligent tutor curricula. However, it does not appear that these sys- 
tems address the root causes of gaming; instead these systems only alleviate 
gaming’s symptoms. Hence, it seems quite possible that students will over 
time discover ways to defeat these interventions over time. 

To have confidence that a redesign will address the root causes of gam- 
ing behavior, rather than briefly alleviating its symptoms, our community 
needs to understand why students game — both in terms of context (what spe- 
cific situations students choose to game in), and of student characteristics 
(what factors differentiate the students who engage in gaming behavior from 
the majority of students who do not). There is already preliminary evidence 
about when students game, gathered in the process of developing detectors 
of gaming behavior, showing that students game on steps they personally 
find difficult. However, knowing that some students game on steps they per- 
sonally find difficult does not tell us why the majority of students do not. 

In this article, we will investigate why some students choose to game while 
other students choose not to game, using evidence from three studies across 
two different types of interactive learning environments, Cognitive Tutors 
(Anderson, Corbett, Koedinger, & Pelletier, 1995) and ASSISTments (Razzaq 
et al., 2007). By investigating this question across multiple systems with inde- 
pendent methods of detecting gaming, we can have some confidence that our 
findings will generalize beyond just a single type of learning environment. 

This article is organized as follows: first, we will discuss a set of hypothe- 
ses for why some students game, drawn from the literature on gaming, moti- 
vation, and related classroom behaviors. Next, we will present evidence on 
teachers’ perspectives about why students game. Then we will discuss three 
questionnaire studies where we gave questionnaire items to students (rele- 
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vant to the hypotheses) and correlated students’ responses to their frequen- 
cy of gaming. Next, we consider how each of the hypotheses is confirmed 
or disconfirmed by the data from the students’ responses, and how the results 
from our studies correspond to the teachers’ predictions. Finally, we discuss 
the psychological and design implications of our results. 


Systems Studied 


Within this article, we will consider data on the student characteristics 
associated with gaming the system from two different interactive learning 
environments: Cognitive Tutors (shown in Figure 1) and ASSISTments 
(shown in Figure 2). Both environments can broadly be characterized as 
intelligent tutoring systems or coached practice, but differ in many ways at 
a finer level of detail. Within each type of learning environment, each stu- 
dent works individually with the computer software to complete mathemat- 
ics problems. Problems in each Cognitive Tutor curriculum are designed to 
map to specific parts of state-mandated mathematics curricula, and are orga- 
nized into lessons by curricular themes. Problems in the ASSISTments sys- 
tem are designed to map to the problems found in a state mathematics exam, 
the Massachusetts Comprehensive Assessment System (Razzaq et al., 2007), 
are explicitly modeled on problems from exams in previous years, and are 
also grouped into curriculums by mathematical topic. 

A Cognitive Tutor breaks down each mathematics problem into the steps 
used to solve it, making the student’s thinking visible, whereas ASSIST- 
ments provide a complete problem, and break down the problem into steps 
only if the student makes errors or has difficulties. 

In both environments, as a student works through a problem, a running 
cognitive model assesses whether the student’s answers map to correct 
understanding or to a known misconception (Anderson et al., 1995), and 
errors are flagged. In addition, each system offers further learning support. 
Cognitive Tutors give tailored feedback when a student’s answer is indica- 
tive of a known misconception. The ASSISTments system responds to a 
wrong answer with both targeted feedback and supplementary questions that 
break down the problem-solving exercise into the required component skills. 

Beyond instant feedback, both systems offer multi-level on demand hints 
to students. When a student requests a hint (by clicking a button), the soft- 
ware first gives a conceptual hint. The student can then request further hints, 
which become more and more specific until the student is given the answer 
(as in Figure 1). The hints are context-sensitive and tailored to the exact 
problem step the student is working on. 

Within a Cognitive Tutor, as the student works through a set of problems, the 
system uses Bayesian Knowledge Tracing (Corbett & Anderson, 1995) to deter- 
mine how well the student is learning the component skills in the cognitive 
model, calculating the probability that the student knows each skill based on that 
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ASSISTMENT PREVIEW - Microsoft Internet Explorer 
File Edt View Favorites Tools Help 
Address |€) http://nthé.wpl.edu/builder /Preview,.do 


2003, Mathematics - Grade 8 
Question 19: Short Answer 
Geometry 


Triangles ABCand DEF shown below are congruent. 


oe, 


The perimeter of AABCis 23 inches. What is the length of side DF in ADEF? 


he original question 
Tnangles ABC and DEF are congruent. <P iat ae 
The perimeter of triangle ABC is 1] a. Congruence 


23 inches. What is the length of b. Perimeter 
side DF in triangle DEF? 





c. Equation -Solving 
Hram, no. 
Let me break this down for you. 


The 1* scaffolding question 


Which side of triangle ABC has the same length (5 


as side DF of tangle DEF? 





What is the perimeter of triangle ABC? The 2 "4 scaffolding question 
O2+x+8 
@) ths * Sy 
©, * x(2x) 

A buggy message 


No. You might be thiiking that the area is 1/2 base times height, but you are looking 
for the perimeter 


Perimeter is defined as the sum of all sides of a figure 


A hint message 





® Done @® internet 
Figure 2. An ASSISTment where a student made an error on the question, 
completed the Ist scaffolding question, and is in the middle of 
trying to answer the 2nd question 
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student’s history of responses within the tutor. Using these estimates of student 
knowledge, the tutoring system gives each student problems, which are relevant 
to the skills which he or she needs to learn. The ASSISTments system does not 
have similar functionality, and instead selects problems from predefined sets. 
Both Cognitive Tutors and the ASSISTments system are used in combi- 
nation with regular classroom instruction; students use the Cognitive Tutor or 
ASSISTments system once or twice a week as part of a regular mathematics 
course, and have classroom lecture or group work on the other days. Both 
systems have been validated to result in positive student learning outcomes. 
Cognitive Tutor curricula have been validated to be highly effective at help- 
ing students learn mathematics, leading not only to better scores on the Math 
SAT standardized test than traditional curricula (Koedinger, Anderson, 
Hadley, & Mark, 1997), but also to a higher percentage of students choosing 
to take upper-level mathematics courses (Carnegie Learning, 2002). In recent 
years, Cognitive Tutor mathematics curricula have come into use in an 
increasing percentage of U.S. high schools — about 6% of U.S. high schools 
as of the 2005-2006 school year. The ASSISTments system is considerably 
newer, and has thus been less thoroughly studied, but its use has also been 
shown to result in significantly better learning (Razzagq et al., 2007). In 2006- 
2007, 3,000 students are using ASSISTments as part of their math class. 


HYPOTHESES 


To determine what factors might lead students to game the system, we 
conducted a thorough review of the relevant literature on gaming-like 
behavior in both traditional classrooms and in online settings. In general, the 
existing literature had considerably more hypothesis than concrete data 
about why students engage in gaming-like behavior, though notable excep- 
tions to this trend do exist (Arbreton, 1998). We also reviewed literature on 
the range of attitudes, beliefs, and motivations students show towards tech- 
nology and school, and the range of attitudes and beliefs that cause people 
to behave in resistant and subversive ways outside of educational settings, 
as will be discussed in the following sections. Finally, we engaged in both 
structured and nonstructured brainstorming (Kelley & Littman, 2001) with 
school teachers and colleagues in a variety of scientific areas, including edu- 
cational technology researchers, educational psychologists, behavior modi- 
fication researchers, and interaction designers. 

It is important to note that not all of this literature review and brain- 
storming took place at once; instead, it took place across a span of three 
years. Some hypothesis formation took place before any of the three studies 
were conducted, but hypothesis formation continued in between each of the 
studies reported in this article. Through this process of discussion, brain- 
storming, and literature review, we have come to 13 hypotheses for why stu- 
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dents game. It is important to note that this is by no means an exhaustive list 
of the reasons students may elect to game the system. A virtually limitless 
set of hypotheses may be generated; we claim only that the set presented is 
reasonably well-justified by either prior research or practitioners’ and 
researchers’ past informal experience. 

For presentation purposes we categorize our 13 hypotheses into groups. 
These categories are meant only to group hypotheses together that have some 
relation to one another, to facilitate understanding, not to make any broader 
theoretical claims. Within this article, the important unit of analysis is the 
hypotheses themselves and their corresponding results, not the categories. 
The five categories of hypotheses are: hypotheses relating gaming to stu- 
dents’ goals (H1, H2), hypotheses relating gaming to students’ attitudes (H3- 
H5), hypotheses relating gaming to students’ beliefs (H6-H9), hypotheses 
relating gaming to students’ broader responses to educational situations (H10, 
H11), and hypotheses relating gaming to students’ emotions (H12, H13). 


Hypothesis H1: Performance Goals 

Our first hypothesis, and the most commonly proposed hypothesis for 
why students game, prior to the research presented here (Baker, Corbett, 
Koedinger, & Wagner, et al., 2004; Martinez-Miron, du Boulay, & Luckin, 
2004), is that students game the system because they have performance 
goals rather than learning goals (the distinction between performance goals 
and learning goals is discussed in detail in Dweck, 2000). In this case, a stu- 
dent using educational software wants to “succeed” in the environment, by 
completing all of the problems, more than he or she wants to learn the mate- 
rial contained in those problems. To complete more problems, the student 
engages in gaming behaviors. 

Some evidence on student behavior in traditional classrooms supports this 
hypothesis. Specifically, some students in traditional classrooms engage in a 
behavior termed “executive help-seeking.” In this behavior, a student starting 
work on a new problem requests help from their teacher or a teacher’s aide 
immediately, before attempting to solve the problem on their own. Arbreton 
(1998) found that executive help-seeking was significantly correlated with 
performance goals, as measured through questionnaire items. 


Hypothesis H2: Desire For More Control 

Our second hypothesis is that students game the system out of a desire for 
more control over their learning experience. In this case, a student using a 
fairly constrained learning environment, such as Cognitive Tutors or 
ASSISTments, feels that they do not have control over their learning expe- 
rience, and games the system to regain control over their learning experience 
(for example, avoiding problems the student does not wish to solve). 
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This hypothesis was formed based on informal comments made by stu- 
dents in previous studies, and is potentially congruent with prior studies that 
have found that giving students greater control within constrained learning 
environments, even over relatively minor aspects of their learning experi- 
ence, can improve student motivation and even increase student learning 
(Cordova & Lepper, 1996). It is possible that one of the reasons choice 
improves learning is because it reduces gaming. 


Hypothesis H3: Disliking Mathematics 

Our third hypothesis is that students game the system in a learning envi- 
ronment because they dislike the subject matter taught in that learning envi- 
ronment — in the case of the specific systems studied within this article, 
mathematics. In this hypothesis, a student who dislikes mathematics 
attempts to make progress without having to engage with the disliked mate- 
rial — by gaming the system. This hypothesis views gaming as a form of 
effort withdrawal, and is consistent with prior research showing that stu- 
dents generally put more effort into learning material that they find interest- 
ing (Schiefele, 1991). 


Hypothesis H4: Disliking Computers 

Our fourth hypothesis is that students game the system because they dis- 
like computers. Potentially, a student who dislikes computers might game 
the system as a means of refusing to work with a computer he or she dis- 
likes, without attracting the teacher’s attention. Alternatively, a student who 
dislikes computers might lack patience for working through problems with 
a computer (and therefore choose to game the system). In general, if a rela- 
tionship exists between disliking computers and gaming the system, it may 
be mediated by some of the beliefs about computers that form hypotheses 8 
and 9, or even by lack of self-drive (hypothesis 12). 


Hypothesis H5: Disliking the Learning Environment 

Our fifth hypothesis is that students game the system in an interactive 
learning environment because they dislike that specific learning environ- 
ment (but not necessarily computers in general). A student might dislike a 
learning environment for many reasons, from disliking features in the envi- 
ronment’s design (see hypothesis 8), to preferring normal classroom work to 
using that learning environment, to disliking having to put in continual effort 
(which may be more difficult to avoid in computer tutors than traditional 
classrooms). Disliking a specific learning environment might lead a student 
to game the system for many of the same reasons that disliking computers 
would, and may be mediated by some of the same factors: beliefs about 
computers (hypotheses 8 and 9), a desire for control (hypothesis 2), or lack 
of self-drive (hypothesis 12). 
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Hypothesis H6: Believing Mathematics is not Important 


Our sixth hypothesis is that students game the system in an interactive 
learning environment because they do not believe that the subject matter 
(mathematics) taught in that learning environment is important. 

This hypothesis is compatible with the prior finding that students put 
more effort into learning material that they personally find important 
(Schiefele, 1991). Many students do not consider mathematics to be person- 
ally important or valuable to their long-term goals (Pettitt, 1995); such stu- 
dents may game the system as a way of reducing their effort. 


Hypothesis H7: Believing Success in Mathematics is Due to Innate 
Ability, not Effort 


Our seventh hypothesis is that students game the system because they 
believe that they will not be able to succeed in the tutor whether or not they 
put in effort. A student may believe this because they believe that some stu- 
dents are good at math and others are bad at math (an entity theory of intelli- 
gence — Stipek & Gralinski, 1996; Dweck, 2000), and worries that they them- 
selves are not good at math. Entity theories have been shown to be associat- 
ed with self-handicapping behaviors (Rhodewalt, 1994; Dweck), suggesting 
that a student with an entity theory may game the system to avoid the expe- 
rience of trying to solve the tutor problems appropriately and failing. 


Hypothesis H8: Believing the Tutor is Not Helpful For Learning 


Our eighth hypothesis is that students game the system because they 
believe that the tutor will not help them learn, perhaps due to dissatisfaction 
with some aspect of the tutor (the design of problems or hint messages, for 
example). Since the student believes they can not learn from the tutor, they 
adopt the strategy of gaming the system to complete problems with minimal 
effort. This hypothesis corresponds to the finding the adoption of software 
generally is strongly associated with how beneficial the user believes the 
software will be (Venkatesh & Davis, 2000). 


Hypothesis H9: Believing Computers/the Tutor are Uncaring 


Our ninth hypothesis is that students game the system because they 
believe that the tutor and/or computer does not (or cannot) care if they learn, 
and thus do not feel motivated to put the same effort into their interactions 
with the tutor as they would with a human teacher or tutor. This hypothesis 
is effectively the converse of the hypothesis by Bickmore and Picard (2004), 
that machines that act in more explicitly caring fashions will improve user 
motivation and performance. If a more caring computer may improve moti- 
vation and performance, it is reasonable to expect that a computer perceived 
as uncaring may worsen motivation and performance. 
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Hypothesis H10: Lack of Educational Self-Drive 

Our 10" hypothesis is that students game the system because they lack 
self-drive, especially for completing tasks, which do not personally motivate 
them. People vary in their degree of patience for completing tasks, which 
they find personally uninteresting or unmotivating (Sansone, Wiebe, & Mor- 
gan, 1999). Though students generally find Cognitive Tutors motivating 
(Schofield, 1995), it is likely that many students find working through math- 
ematics problems in computer software unmotivating to at least some 
degree, or in some situations. Which students persist in working through 
problems carefully, and which students choose to game the system, may 
depend on students’ general orientation to educational situations. 

The construct of self-drive is strongly related to conscientiousness 
(McCrae & Costa, 1990; Sansone et al., 1999); we use the term self-drive 
here simply to distinguish lack of self-drive in specific educational settings 
from the more global construct that conscientiousness is thought to be. In line 
with that goal, the items used in Tutor Study 2 to assess this construct are 
drawn from a study skills inventory rather than from a personality inventory. 


Hypothesis H11: Passive-Aggressiveness 

Our 11" hypothesis is that gaming the system is a type of passive-aggres- 
sive behavior. Passive-aggressive behavior is defined as a “pervasive pattern 
of negativistic attitudes and passive resistance to demands for adequate per- 
formance” (American Psychiatric Association, 1994, pp.634-635), and is 
associated with behaviors such as procrastination, argumentativeness, 
obstructive behavior, and pretending to forget obligations (Stone, 1993). 
Hence, under this hypothesis, a student games the system specifically to 
resist using the software to learn mathematics (as the teacher requests) with- 
out visibly appearing to be resisting. 

It is important to note that we are not necessarily saying that students who 
game the system do so because of personality disorders. However, the over- 
all construct of passive-aggressiveness may relate to a student’s decision to 
game the system. 


Hypothesis H12: Frustration 


Our 12" hypothesis is that students game the system when and/or because 
they feel frustrated by the tutoring software. In this hypothesis, a student 
feels frustrated because of the difficulty of the mathematics or the challenges 
of using the tutoring software, and games the system to bypass the frustrat- 
ing material or interface issue. 

Prior research appears to suggest that frustration can lead to lack of persis- 
tence in using an interactive system, both within educational software and other 
types of software — or at least that alleviating frustration significantly improves 
persistence. A wizard-of-Oz study of affective scaffolding in a reading tutor 
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found that students expressed considerable frustration with errors in the tutor’s 
voice recognition, and addressing that frustration caused students to persist in 
using the software longer (Aist, Kort, Reilly, Mostow, & Picard, 2002). 


Hypothesis H13: Anxiety 


Our 13", and final, hypothesis is that students game the system because 
they feel anxious when they use the tutor (perhaps because they find the 
mathematics difficult or believe other students are performing better in the 
tutor than they are). Gaming may both enable a student to appear to succeed 
(by completing problems successfully over time) without risking failure, and 
provide a student with an excuse for not knowing the material — the student 
does not know the material because he or she did not try not because the stu- 
dent tried and failed — a behavior known as self-handicapping (Rhodewalt, 
1994). Anxiety and self-handicapping behavior have been found to be relat- 
ed to one another in past research (Dweck, 2000). 


TEACHER PERSPECTIVES 


Teachers play an essential role in intelligent tutor classrooms (Schofield, 
1995). Since teachers observe students using intelligent tutors for consider- 
able periods of time, they may have useful insight as to why students choose 
to game. We sought to capture teacher perspectives, and compare them to the 
results found by analyzing students’ self-reports. These results should be 
useful in developing systems that respond to gaming in a way that teachers 
find valid and appropriate, so that our systems and the teachers will not work 
at cross-purposes. In this section we report the results of a survey designed 
to determine why teachers think students game the system. 


Methodology 


We surveyed 18 seventh and eighth grade public school teachers who had 
used ASSISTments in their classes about why they thought students “game 
the system.” Thirteen (13) of the teachers had used the system for over a year. 

The teachers were given a definition of gaming in the survey instructions 
(shown in the Appendix). After reading our definition of gaming behavior, 
all teachers understood it well enough to answer the survey — none of the 
teachers reported not understanding what gaming was. 

In the survey, we asked the teachers to rank our 13 hypotheses by mark- 
ing the top three reasons they thought students gamed (with /, 2, or 3), and 
marking any hypothesis that they thought was absolutely not responsible for 
or related to student gaming with an_X. To avoid biasing our results with the 
presentation order we grouped our 13 hypotheses into the same five general 
categories used for presentation in the Hypotheses section, and made five 
different versions of the survey form, each with a different ordering of the 
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five sections. In addition, a space was provided on the form for teachers to 
provide other explanations for why students game, to generate new hypothe- 
ses. One version of the survey is reproduced in the Appendix. 

One potential flaw with our methodology was that the survey was adminis- 
tered by one of the creators of the ASSISTment system (the third author), who 
was personally known by the teachers. This may have discouraged the respon- 
dents from blaming gaming directly on flaws in the ASSISTment system. 


Results 

Table 1 shows the study’s results. The first row of data shows how many 
teachers ranked each hypothesis as the most important reason why students 
game. The second data row shows the number of teachers that ranked each 
hypothesis as second most likely, and the third data row shows the third- 
place rankings. The “Sum” column aggregates the total number of votes in 
favor of each hypothesis. The row called “Disagreements” shows the num- 
ber of teachers that explicitly stated that a particular hypothesis would not 
explain why students game. 

Three explanations were significantly more popular than the rest: Frus- 
tration (H12), Lack of Self-Drive (H10), and Performance Goals (H1). At 
least half of the 18 teachers listed these three reasons; no other hypothesis 
was selected by more than five of the teachers. Frustration (H12) was the 
clear favorite explanation, listed as a top-three explanation by the largest 
number of teachers, and listed as the most important explanation by the 
largest number of teachers (9 of the 18 teachers). 

Five hypotheses were particularly unpopular: H4 (Disliking Computers), 
H2 (Desire for Control), H5 (Disliking the Learning Environment), H9 
(Believing Computers/the Tutor are Uncaring), and H8 (Believing the Tutor 
is Not Helpful For Learning). None of these hypotheses had a single vote in 
favor, and was explicitly rejected by 7 or more of the 18 teachers. From this 
result, we can conclude that teachers did not think that students’ negative 
attitudes towards computers or the learning environment were central rea- 
sons for gaming. 

In addition, many open-response comments about gaming seemed to be 
related to effort avoidance: 


¢ “they are not trying hard,” 

° “lack of effort,” 

¢ “do not want to read,” 

* “too long to go through the hints,” 
¢ “instant gratification,” and 


rz 


« “some students may game because they lack motivation to learn inde- 
pendently.” 


Baker, Walonoski, Heffernan, Roll, Corbett, and Koedinger 
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Two other comments focused on students’ lack of math background: 
« “don’t know the math,” and 


« “T would guess that students who have the most difficulty with the ques- 
tions game the most. Many of my 7th grade students do not have 7th 
grade math ability. They give up, due to lack of understanding, assume 
they cannot learn it and begin guessing/gaming.” 


Another interesting comment made by multiple teachers was that many 
students game the system in the classroom on days when they are not using 
the computer, as in Arbreton (1998). 

We conclude that teachers believe that students game the system because 
they are frustrated, and/or lack the drive to put forth the effort. Teachers also 
believed that student gamed because they had performance goals. 


QUESTIONNAIRE STUDIES 


In three separate studies, questionnaires were given to students, with 
questions relevant to the 13 hypotheses about gaming. By correlating the 
incidence of gaming the system (detected using behavior detectors devel- 
oped separately for each system — those detectors will be described in this 
section) to student responses on the questionnaires, it is possible to assess to 
what degree each of the hypotheses appear to be associated with gaming. 

The first study conducted on why students game, Cognitive Tutor Study 
One, was conducted in Spring 2004. Cognitive Tutor Study Two and 
ASSISTments Study were conducted after Cognitive Tutor Study One had 
been presented at a conference (Baker et al., 2005), and their designs were 
influenced by that study’s results. Cognitive Tutor Study Two was designed 
to refine the results in Cognitive Tutor Study One and investigate hypothe- 
ses not considered in that study. The ASSISTments Study was intended both 
to replicate Cognitive Tutor Study One in a different learning environment, 
and to investigate hypotheses not considered in that study. This section 
describes the methods for all three studies and the next section reports the 
results for the three studies. 


Cognitive Tutor Study One 


Study goals. The first study on the relationship between student goals, 
beliefs, and motivations, and the choice to game the system, was designed 
primarily to investigate hypothesis H1 (performance goals) and hypothesis 
H13 (anxiety). Hypothesis Hl was the hypothesis most strongly predicted 
by the literature and the research community (Baker, Corbett, Koedinger, & 
Wagner, 2004; Martinez-Miron et al., 2004); hypothesis H13 was the 
hypothesis most strongly predicted by teachers collaborating with the Cog- 
nitive Tutor research group, in informal conversations. Hypotheses H4 (dis- 
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liking computers) and H5 (disliking the interactive learning environment) 
were also considered within this study. 


Participants. These four hypotheses for why students choose to game the 
system were studied within the context of six classes at two schools in the 
Pittsburgh suburbs. All students were in the second semester of a year-long 
cognitive tutor course for middle school mathematics, in 2004. All students 
in the study had used Cognitive Tutors multiple times a week for several 
months. This study took place within a short (two class period) Cognitive 
Tutor lesson on scatterplot generation and interpretation. Student ages 
ranged from approximately 12 to 14. One hundred two (102) students com- 
pleted all stages of the study; 23 other students were removed from analysis 
due to missing one or more parts of the study. 


Cognitive tutor curriculum. In Cognitive Tutor Study One, students were 
studied as they used a cognitive tutor lesson on scatterplot generation and 
interpretation (shown in Figure 1), which was known from past research to 
lead to significantly improved learning. Full detail on this lesson is given in 
Baker (2005). 


Gaming behavior model. The incidence of gaming behavior, for each stu- 
dent, was assessed using a detector of gaming behavior that identifies how 
often each student is gaming the system. This detector is discussed in detail 
in Baker (2005); in this section we review of some of the detector’s essen- 
tial details. 

The cognitive tutor gaming detector was designed using a psychometric 
modeling technique, Latent Response Models (Maris, 1995). This frame- 
work makes predictions at two levels: whether or not a specific action is 
gaming, and how often each individual student games. This multi-level 
modeling framework gives the model extra sensitivity in using all of the 
available data on the prevalence of gaming, across students. 

The cognitive tutor gaming detector was trained on data from a combi- 
nation of systematic classroom observations of whether a student was gam- 
ing (Baker, Corbett, Koedinger, & Wagner, 2004) and data about the stu- 
dent’s pretest and posttest scores. The gaming detector is significantly better 
than chance at distinguishing students who game the system from nongam- 
ing students, and is also significantly better than chance at distinguishing 
students who game the system from students who display gaming-like 
behaviors as part of a broader pattern of focusing time and energy on the 
most challenging material (Baker, Corbett, & Koedinger, 2004; Baker, Cor- 
bett, Koedinger, & Roll, 2006). The cognitive tutor gaming detector search- 
es for fast errors and help requests made on the same step across multiple 
problems, particularly when the student has never demonstrated proficiency 
on that step. 
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Questionnaire. The questionnaire consisted of a set of self-report questions 
given along with the students’ unit pretest, to assess students’ motivations 
and beliefs. The questionnaire items were drawn from existing motivational 
inventories or from items used across many prior studies with this age group 
(as noted next, per question), and were adapted minimally (for instance, the 
words “the computer tutor” was regularly substituted for “in class,” and 
questions were changed from first-person to second-person for consistency). 
All items were pretested for comprehensibility with a student from the rele- 
vant age group before the study. Items were a combination of multiple 
choice and Likert-scale items, from 1 (Not at all) to 6 (Very much). 


The questionnaire included items to assess: 
¢ Hypothesis H1: Whether the student was oriented towards perfor- 
mance or learning (multiple choice; adapted from Mueller & Dweck, 1998) 
“We are considering adding a new feature to the computer tutors, to give 
you more control over the problems the tutor gives you. If you had your 
choice, what kind of problems would you like best? 
A) Problems that aren’t too hard, so I don’t get many wrong. 
B) Problems that are pretty easy, so I'll do well. 
C) Problems that I’m pretty good at, so I can show that I’m smart 
D) Problems that Ill learn a lot from, even if I won’t look so smart.” 
“Some classes that use computer tutors also have extra-credit projects. If you 
had your choice, what kinds of extra projects would you most like to do? 
A) An extra-credit project that is easy, so I can get a better grade. 
B) An extra-credit project where I could learn about things that inter- 
ested me. 
C) An extra-credit project in an area I’m pretty good at, so I can show 
my teacher what I know. 
D) An extra-credit project that isn’t very difficult, so I don’t have to 
work too hard. 


¢ Hypothesis H13: The student’s level of anxiety about using the tutor 
(Likert scale) (adapted from Harnisch, Hill, & Fyans, 1980) 

“When you start a new problem in the tutor, do you feel afraid that you 

will do poorly?” 

“When you are working problems in the tutor, do you feel that other stu- 

dents understand the tutor better than you?” 


¢ Hypothesis H13: The student’s level of anxiety about using com- 
puters (Likert scale) (Harnisch, Hill, & Fyans, 1980) 

“When you use computers in general, do you feel afraid that you will do 

something wrong?” 
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¢ Hypothesis H5: How much the student liked using the tutor (Likert 
scale) (adapted from Mueller & Dweck, 1998) 

“How much fun were the math problems in the last computer tutor lesson 

you used?” 

“How much do you like using the computer tutor to work through math 

problems?” 


¢ Hypothesis H4: The student’s attitude towards computers (Likert 
scale) (Frantom, Green, & Hoffman, 2002) 


“How much do you like using computers, in general?” 


¢ Whether the student was lying or answering carelessly on the ques- 
tionnaire (2 choices) (Sarason, 1978) 

“Ts the following statement true about YOU? ‘I never worry what other 

people think of me.” TRUE/FALSE” 


Results from this study will be presented, along with the results of the 
other two studies, in section 5. 


Cognitive Tutor Study Two 


Study goals. The second study on the choice to game a Cognitive Tutor 
investigated several hypotheses including hypotheses H2 (desire for con- 
trol), H3 (disliking mathematics), H8 (the belief that the tutor is not helpful 
for learning), H9 (the belief that computers/the tutor are uncaring), H10 
(lack of educational self-drive), and H11 (passive-aggressiveness). This 
study was conducted in Spring 2005, after Cognitive Tutor Study One was 
analyzed, and was designed with the results of that study in mind (the results 
of both studies will be discussed in the next section). 


Participants. Cognitive Tutor Study Two was conducted in five classes at 
two schools in the Pittsburgh suburbs. All students were in the second 
semester of a year-long cognitive tutor course for middle school mathemat- 
ics, and had used Cognitive Tutors multiple times a week for several months. 
Student ages ranged from approximately 12 to 14. One hundred eight (108) 
students completed all stages of the study; 13 other students were removed 
from analysis due to missing one or more parts of the study. 


Cognitive tutor curriculum. Cognitive Tutor Study Two involved two tutor 
lessons: the lesson on scatterplot generation and interpretation investigated 
in Cognitive Tutor Study One, and a lesson drawn from the same curricu- 
lum, on converting between percents and other mathematical representations 
(shown in Figure 3). Both lessons had been verified in earlier work to lead 
to significantly improved learning, on average. More detail on both lessons 
is given in Baker (2005). Forty-seven percent (47%) of the students used the 
scatterplot generation and interpretation lesson and the other 53% of the stu- 
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dents used the lesson on converting between percents and other mathemati- 
cal representations. Both lessons were used by students for 80 minutes. 


Gaming behavior model. The same model of gaming behavior used in Cog- 
nitive Tutor Study One was used to assess each student’s frequency of gam- 
ing behavior within Cognitive Tutor Study Two. The model has been shown 
to be effective in both lessons (Baker, Corbett, Koedinger, & Roll, 2006). 


Questionnaire. The questionnaire in Cognitive Tutor Study Two consisted of 
a set of self-report questions given along with the students’ unit pretest, to 
assess students’ motivations and beliefs. The questionnaire items were drawn 
from existing motivational inventories or from items used across many prior 
studies with this age group; some were adapted minimally, to shift their 
domain to the context of a tutoring system. All items were pretested for com- 
prehensibility, but some students needed an explanation of the item “I feel 
that the tutor, in its own unique way, is genuinely concerned about my learn- 
ing,” during the actual study, because of difficulty interpreting the word “con- 
cerned” in this context. All questionnaire items in this study were given as 
Likert scales, from | (Strongly Disagree) to 6 (Strongly Agree). 


The following items were used in the questionnaire. 


¢ Hypothesis H2 (desire for more control, expressed in reverse as the 
belief that computers/the tutor increase control) 


“Using the tutor gives me greater control over my work.” (Dillon, Gar- 
ner, Kuilboer, & Quinn, 1998) 
“T am in complete control when I use a computer.” (Selwyn, 1997) 


¢ Hypothesis H3 (disliking mathematics) 
“T enjoy working on a difficult math problem.” 
“Math is boring.” (Knezek & Christensen, 1995) 


¢ Hypothesis H8 (believing the tutor is not useful) 

“The tutor’s help system is effective in helping me complete problems.” 
(Lewis, 1995) 

¢ Hypothesis H9 (believing computers/the tutor are uncaring) 


“T feel that the tutor, in its own unique way, is genuinely concerned about 
my learning.” (Bickmore & Picard, 2004) 


“The tutor treats people as individuals.” 
“The tutor ignores my feelings.” (Cupach & Spitzberg, 1983) 


¢ Hypothesis H10 (lack of self-drive) 
“T study by myself without anyone forcing me to study.” 
“T try to finish whatever I begin” (Knezek & Christensen, 1995) 
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¢ Hypothesis H11 (tendency towards passive-aggressiveness) 
“At times I tend to work slowly or do a bad job on tasks I don’t want to do.” 
“T tend to try to get out of things by making up excuses.” 


“I often forget things that I would prefer not to do.” (Parker & Hadzi- 
Pavlovic, 2001) 


¢ Additional item (initially related to a broader version of Hypothesis H8) 


“Most things that a computer can be used for, I can do just as well 
myself.” (Selwyn, 1997) 


Results from this study will be presented, along with the results of the 
other two studies, in the next section. 


ASSISTments Study 


Study goals. The third study on why students choose to game was also con- 
ducted in Spring 2005, at approximately the same time as Cognitive Tutor 
Study Two. The research in the third study was designed to address ques- 
tions raised by Cognitive Tutor Study One, as well as address hypotheses not 
considered in that study. It was tailored to the ASSISTments System (Raz- 
zaq et al. 2007) and was designed to investigate several hypotheses, includ- 
ing H1 (Performance Goals), H3 (Disliking Mathematics), and H4 (Dislik- 
ing Computers), H5 (Disliking the Learning Environment), H6 (Believing 
Mathematics is Not Important), H7 (Believing Mathematics Ability is 
Innate), H8 (Believing the Tutor is Not Helpful for Learning), H10 (Lack of 
Educational Self-Drive), H11 (Passive-Aggressiveness), and H12 (Frustra- 
tion). The results of the study, along with the results of the two Cognitive 
Tutor studies, will be discussed in the Results section. 


Participants. Data for these hypotheses was collected by way of a survey 
administered to students who had been regularly using the ASSISTments 
system throughout the 2004-2005 academic year, on a biweekly basis. The 
students were from two urban schools and multiple classes (eight teachers) 
in Worcester, Massachusetts. About 600 students were using the system in 
2004-2005. Only students who completed all stages of the study were 
included in the final dataset, which included data and responses for 323 stu- 
dents (Walonoski and Heffernan 2006a). 


ASSIS Tments curriculum. Unlike the studies conducted with the Cognitive 
Tutor curriculum, which were conducted using specific subsegments of the 
Cognitive Tutor curriculum, the studies conducted with the ASSISTments 
system involved problems drawn from the entire ASSISTments curriculum, 
involving data from an entire academic year drawn over many sets of prob- 
lems grouped by mathematical topics. 
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Gaming behavior model. In the ASSISTments study, each student’s fre- 
quency of gaming behavior was calculated with a knowledge-engineered 
model of gaming behavior. The model calculated how frequently each stu- 
dent gamed based only on knowledge-engineered definitions of hint abuse 
and guessing-and-checking behavior (Walonoski & Heffernan, 2006b). The 
model only considered potential gaming behaviors that occur in clusters, 
based upon the finding from the Cognitive Tutor gaming model that gaming 
actions occur in clusters (Baker, Corbett, & Koedinger, 2004). Using the 
model, gaming rates were calculated for each student who completed an 
end-of-year survey. 

We used this knowledge-engineered model to analyze our hypotheses 
within the ASSISTments study data, providing a potential contrast to the 
machine-learned model used to analyze the hypotheses in the Cognitive 
Tutor study data. It has been shown to be possible to develop a machine- 
learned model of gaming behavior within the ASSISTments data (Walonos- 
ki & Heffernan, 2006a). However, using a different detection approach with 
our ASSISTments study data considerably lowers the risk of bias in our 
analyses. The existing machine-learned model of gaming behavior in the 
ASSISTments system was developed using data features and a modeling 
framework heavily influenced by the earlier design of the Cognitive Tutor 
gaming detector. Hence any bias incorporated into the machine-learned 
model used to analyze the Cognitive Tutor data is likely to also be contained 
in a machine-learned model of gaming within the ASSISTments detector. In 
this way, if a consistent pattern of results is seen across the two systems, we 
will know that the pattern of results is inherent to gaming behavior rather 
than arising from biases in our detection method. 


Questionnaire. The ASSISTments survey consisted of 22 Likert-scale 
items, one Yes/No question, and two open response items. The Likert-scale 
items were ona I-5 scale, with the options “Strongly Disagree,” “Disagree,” 
“Not Sure,” “Agree,” and “Strongly Agree.” The survey questions were not 
developed to map in an exclusive one-to-one fashion to a list of hypotheses 
(unlike the Cognitive Tutor studies); hence, some of the items correspond to 
more than one hypothesis, and not all hypotheses are represented by an 
equal number of items. The mapping between hypotheses and items pre- 
sented here was developed through concept mapping exercises and discus- 
sion between the authors. 

The survey questions used in the ASSISTments study were as follows 
(Table 2). 

Our statistical and quantitative analysis of this survey was restricted to 
the 22 Likert-scale questions and the single Yes/No question. The open- 
response questions were not classified or coded, since the analysis would 
necessarily be quite complex and time-consuming. However, rather than 
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omit this data source entirely, we will discuss some of the open-responses in 
the next section, to illustrate the quantitative findings. 

Results from the ASSISTments study will be presented, along with the 
results of the other two studies, in the Results section. 


RESULTS: EVIDENCE ON HYPOTHESES 


In this section we will discuss the evidence in favor of and against each 
hypothesis. Evidence for and against each hypothesis is obtained by corre- 
lating the answers on the relevant questionnaire items to each student’s fre- 
quency of gaming behavior, as detected by the models of gaming. 


Student Goals: H1 and H2 


1: Performance goals. Both Cognitive Tutor Study One and the ASSIST- 
ments Study had items relating to the first hypothesis: students game the sys- 
tem because they have performance goals rather than learning goals. Cogni- 
tive Tutor Study One had two multiple-choice questions related to this 
hypothesis. Neither item showed a significant correlation between gaming 
and performance goals. 


« “We are considering adding a new feature to the computer tutors, to 
give you more control over the problems the tutor gives you. If you had 
your choice, what kind of problems would you like best?” 


0 r= -0.03, F(1,100)=0.09, p=0.76 
« “Some classes that use computer tutors also have extra-credit projects. 


If you had your choice, what kinds of extra projects would you most 
like to do?” 
o r=-0.06, F(1,100)=0.41, p=0.52 

The ASSISTments Study had five questions related to the performance 
goal hypothesis. Out of those five questions, four had no significant corre- 
lation. One question was marginally significantly, but weakly, correlated: 

¢ “My goal when using the ASSISTment system was to get through as 

many items as possible.” 
o r=0.10, F(1,323)=2.97, p=0.09. 

Overall, then, there is no strong evidence of a connection between per- 
formance goals and gaming, despite prior predictions that a connection 
would exist (Baker, Corbett, Koedinger, & Wagner, 2004; Martinez-Miron 
et al., 2004). Neither item from Cognitive Tutor Study One was connected 
with gaming, and only one item out of five in the ASSISTments Study was 
even marginally significant, an overall pattern that would occur by chance a 
considerable percentage of the time. Thus, our data from across the two 
studies suggests that students do not game because of performance goals. 
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2: Desire for more control. Only Cognitive Tutor Study Two had elements 
relating to the second hypothesis: students game the system out of a desire 
for more control over their learning experience. 

Cognitive Tutor Study Two had two Likert-scale questions relating to this 
hypothesis. Neither of them was significantly correlated with gaming the 
system. 

¢ “Using the tutor gives me greater control over my work” (Dillon et al., 

1998) 
o r= -0.10, FU,95)=0.92, p=0.34. 
«“T am in complete control when I use a computer” (Selwyn, 1997) 
o r=0.02, F(1,95)=0.04, p=0.85 


Student Attitudes: H3-H5 

H3: Disliking math. Cognitive Tutor Study Two and the ASSISTments 
Study featured elements relating to the third hypothesis: students game the 
system in an interactive learning environment because they dislike the sub- 
ject matter taught in that learning environment. 

In Cognitive Tutor Study Two, the evidence suggested a relationship 
between disliking mathematics and gaming, with both items relevant to that 
relationship pointing (relatively weakly) in that direction. 

«Math is boring” (Knezek & Christensen, 1995) 

0 r=0.19, F(1,95)=3.47, p=0.07 
¢“T enjoy working on a difficult math problem” (Knezek and Chris- 
tensen, 1995) 
0 r= -0.15, FU,95)=2.12, p=0.15 

The ASSISTments Study found stronger evidence that disliking mathe- 
matics was significantly correlated with gaming. The correlations for the 
related questions are as follows: 

¢ “T like math class.” 

0 r= -0.21, F(1,323)=14.89, p<0.001 
¢ “T am good at math.” 

0 r= -0.26, F(1,323)=22.99, p<0.001 
¢ “T work hard at math.” 

0 r= -0.13, F(1,323)=5.7, p=0.02 

Hence, disliking the subject matter (mathematics, in these studies) appears 
to be associated with gaming the system in both learning environments. 

Additional anecdotal support for this conclusion comes from the 
ASSISTments System open response items. About 8% of students who were 
in the top quartile of gamers listed "math" in their dislikes about the system. 
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Another 8% in that quartile said that they liked the system because they 
claimed it helped them learn math. No student whatsoever said they liked 
the system because of “math.” 


14: Disliking computers. Cognitive Tutor Study One and the ASSISTments 
Study both include questionnaire items that are related to the fourth hypoth- 
esis: students game the system because they dislike computers. 

Cognitive Tutor Study One found that students who gamed the system (as 
assessed by the detector) liked computers significantly less than the other 
students. 


¢ “How much do you like using computers, in general?” 
0 r= -0.20, F(1,100)= 4.37, p=0.04 
The ASSISTments Study did not have any items directly addressing this 
hypothesis. One item involved how much students liked learning from a 
computer. This item was not significantly related to gaming. 
¢ “T like learning from a computer.” 
0 r= 0.05, F(1,323)=0.77, p=0.38 
Hence, it appears that there is some evidence that disliking computers is 
associated with gaming, from Cognitive Tutor Study One, but there is no 
confirmation from the ASSISTments Study. 


HS: Disliking the learning environment. Both Cognitive Tutor Study One 
and the ASSISTments Study included items regarding the fifth hypothesis: 
that students game the system because they dislike the learning environment 
(but not necessarily computers in general). 

Within Cognitive Tutor Study One, a student’s attitude towards the tutor 
was significantly correlated to the choice to game the system (marginally in 
one case). 

¢ “How much fun were the math problems in the last computer tutor les- 

son you used?” 
0 r= -0.19, F(1,100)=3.94, p=0.05 

¢ “How much do you like using the computer tutor to work through math 

problems?” 
0 r= -0.16, F(1,100)=2.85, p=0.09 

In relevant items in the ASSISTments Study, gamers were not found to 
be more likely to dislike the system. 

¢ “T liked using the ASSISTment system better than doing my homework.” 

0 r= -0.05, F(1,323)=0.79, p=0.37 
¢ “T liked using the ASSISTment system better than doing a test.” 
0 r= 0.02, F(1,323)=0.14, p=0.70 
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¢ “T liked using the ASSISTment system better than my normal classroom 
activity.” 
o r= 0.00, F(1,323)=0.002, p=0.96 
¢ “Now assume you had internet access at home and that your teacher 
made it possible for you to do your homework either on paper or using 
the ASSISTment system. How much would you prefer to use the 
ASSISTment system?” 
0 r= -0.07, F(1,323)=1.58, p=0.21 
Hence, although the Cognitive Tutor Study found significant correlation and 
support for this hypothesis, there was no such result in the ASSIStments Study. 
One possibility is that students dislike the two systems for different reasons, 
and thus the relationship between disliking the systems and gaming is different. 


Student Beliefs H6-H9 
H6: Math is not important. Only the ASSISTment Study had questions 
relating to the sixth hypothesis: that students game the system because 
they do not believe that the subject matter being taught is important (in this 
case, mathematics). 
The ASSISTments Study had three questions relating (to at least some 
degree) to this hypothesis. None of these items were significantly correlated 
with gaming the system. 
¢ “When I grow up I think I will use math in my job.” 
0 r= -0.05, F(1,323)=0.79, p=0.37 

¢ “T care a great deal about how well I do in math.” 
0 r= -0.08, F(1,323)=2.10, p=0.15 

¢ “My parents think it's very important to do well in math.” 
0 r= 0.03, F(1,323)=0.36, p=0.55 


A7: Believing mathematics ability is innate. Only the ASSISTments Study 
had questions relating to the seventh hypothesis: that gaming the system 
stems from students’ beliefs that mathematics ability is innate, and they will 
not be able to succeed whether they put in effort or not, presumably because 
they do not possess this innate ability. 
¢ “T think that some people are just good at math.” 
0 r= 0.06, F(1,323)=1.31, p=0.25 
* “T believe that if I work hard at math I can do well.” 
0 r= -0.12, F(1,323)=4.49, p=0.04 
These results present mixed evidence on whether believing in mathemat- 
ics as an innate ability is related to students gaming the system — it does 
appear that students who game the system do not believe that hard work will 
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help them with mathematics, presenting evidence in favor of this hypothesis 
— however, the correlation is fairly modest, and not seen in the other item. 


8: The tutor is not helpful for learning. Both Cognitive Tutor Study Two 
and the ASSISTments Study featured questions relating to the eighth 
hypothesis: that students game the system because they believe that the tutor 
will not help them learn. 
In Cognitive Tutor Study Two, the belief that the tutor’s help system was 
not effective was marginally significantly correlated with gaming the system. 
« “The tutor's help system is effective in helping me complete problems.” 
0 r= -0.19, F(1,95)=3.29, p=0.07 
The ASSISTments Study had four questions related to the eighth hypoth- 
esis, of which one was statistically significant. The significant item was 
related to the tutor’s help system; however one nonsignificant item was also 
related to the tutor’s help system. 
¢ “T think that being told the answer was more helpful than reading the hints.” 
0 r= 0.12, FU1,323)=4.61, p=0.03 
¢ “Breaking a question down into smaller steps helped me understand 
how to solve similar problems.” 
0 r= -0.04, F(1,323)=0.51, p=0.47 
¢ “T think the hints helped me understand how to solve similar problems.” 
0 r= 0.07, F(1,323)=1.37, p=0.24 
¢ “T think the ASSISTment system helped me prepare for the MCAS.” 
0 r= -0.04, F(1,323)=0.64, p=0.42 
Hence, in both studies, there was some connection between not believing 
in the tutor’s help systems and gaming the system — but it was only margin- 


ally significant in Cognitive Tutor Study Two, and inconsistent in the 
ASSISTments Study. 


9: Computers/tutors are uncaring. Only Cognitive Tutor Study Two had 
questions related to the ninth hypothesis: students game the system because 
they believe that the tutor and/or computer does not (or cannot) care if they 
learn. The study had three questions related to this hypothesis: 
¢“T feel that the tutor, in its own unique way, is genuinely concerned 
about my learning.” 
0 r= -0.05, F(1,93)=0.25, p=0.62 
¢ “The tutor treats people as individuals” 
0 r= -0.13, F(1,93)=1.53, p=0.22 
¢ “The tutor ignores my feelings” 
0 r= 0.09, F(1,93)=0.85, p=0.36 
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None of these questions had any significant correlation with gaming 
behavior. Hence, there does not appear to be any support for this hypothesis. 


Student Approaches: H10 and H11 

A110: Lack of education self-drive. Both Cognitive Tutor Study Two and the 
ASSISTments Study featured questions relating to the tenth hypothesis: stu- 
dents game the system because they lack self-drive, especially for complet- 
ing tasks which do not personally motivate them. 

Cognitive Tutor Study Two featured two items related to this hypothesis. 
One item had a significant correlation to gaming the system, and the other 
item had a marginally significant correlation. 

¢ “T try to finish whatever I begin.” 

0 r= -0.26, F(1,94)=6.25, p=0.01 
¢ “I study by myself without anyone forcing me to study.” 
0 r= -0.18, F(1,94)=3.05, p=0.08 

The ASSISTments Study had eight questions related to this hypothesis, two 
of which were statistically significant or marginally statistically significant. 

¢ “T work hard at math.” 

0 r= 0.13, F(1,323)=5.7, p=0.02 
¢ “I do my homework in math class.” 
0 r= -0.10, F(1,323)=3.39, p=0.07 
¢ “When a problem was too hard I would just try to get though a problem 
so that I could get into items that I could do.” 
0 r= 0.02, F(1,323)=0.12, p=0.74 
« “T took my work on the ASSISTment system very seriously and worked 
hard all the time.” 
0 r= -0.03, F(1,323)=0.35, p=0.55 
“My goal when using the ASSISTment system was to learn new things.” 
0 r=0.03, F(1,323)=0.24, p=0.62 
¢ “T care a great deal about how well I do in math.” 
0 r= -0.08, F(1,323)=2.10, p=0.15 
¢ “T usually seek help when I don't understand something.” 
0 r= -0.07, F(1,323)=1.56, p=0.21 

Since only two of the eight items relevant to this hypothesis in the 
ASSISTments Study favored this hypothesis, one significantly and the other 
marginally significantly, there is still a 24.5% chance this pattern of results 


within the ASSISTments Study was obtained by chance (computed using the 
properties of probability). 
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Hence, the evidence from Cognitive Tutor Study Two was unambiguous- 
ly in favor of this hypothesis, but the ASSISTments Study’s evidence was 
inconsistent and could be due to chance. 

In the ASSISTment Study’s open response section, many of the students 
in the top quartile of gaming behavior reported getting tired for various rea- 
sons, including too many problems, too many similar problems, too many 
words, and so forth. This reported tiredness might be related to the lack of 
self-drive. 


All: Passive-aggressiveness. Only Cognitive Tutor Study Two had any 
questionnaire items relating to the eleventh hypothesis: that gaming the sys- 
tem is a type of passive-aggressive behavior. 

« “At times I tend to work slowly or do a bad job on tasks I don’t want to do.” 


0 r=0.12, F(1,94)=1.25, p=0.27 


¢ “T often forget things that I would prefer not to do.” (Parker & Hadzi- 
Pavlovic, 2001) 


0 r=0.03, F(1,94)=0.14, p=0.71 
Since neither of these items was significantly correlated with gaming the 


system, there is no evidence that passive-aggressiveness is associated with 
gaming the system. 


Student Emotions: H12 and H13 


A112: Frustration. Only the ASSISTments Study had any questions directly 
related to the twelfth hypothesis: frustration with the tutoring software caus- 
es the students to game the system. 

¢ “T found many of the items frustrating because they were too hard.” 


0 r=0.228, F(1,322)=17.06, p<0.001 


Hence, frustration due to problem difficulty appears to be related to stu- 
dent choice to game the system. 

Anecdotally, in the ASSISTment Study almost every single student in the 
top quartile of gaming expressed in the open response section one of their 
dislikes as being frustrated — more specifically, frustration due to problem 
difficulty, too many words to read, poor wording, and too many problems. 
Other students reported frustration much less often. 


#13: Anxiety. Only Cognitive Tutor Study One had any questions regarding 
the 13th hypothesis: students game the system because they feel anxious 
about using the tutoring software. The study had three questions addressing 
this area: 
¢ “When you start a new problem in the tutor, do you feel afraid that you 
will do poorly?” 


0 r= -0.04, F(1,100)=0.13, p=0.71 
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¢ “When you are working problems in the tutor, do you feel that other stu- 
dents understand the tutor better than you?” 


0 r= -0.03, F(1,100)=0.13, p=0.72 


« “When you use computers in general, do you feel afraid that you will 
do something wrong?” 


0 r= -0.02, F(1,100)=0.05, p=0.82 


None of the responses to these questions were significantly correlated 
with gaming the system. 


Summary of Evidence 


Overall, the results from these three studies appear to be consistent with 
the following hypotheses: 
¢ H3: Dislike of the subject matter 


¢ H10: Lack of educational self-drive 
¢ H12: Frustration 


In addition, the results from these studies appear to be consistent, at least 
partially, with the following hypotheses — although the results in these cases 
are not 100% certain. 

¢ H4: Dislike of computers 


¢ H5: Dislike of the learning environment 
¢ H7: Belief that mathematics ability is innate 
¢ H8: The tutor is not helpful for learning 


Previous hypotheses, including performance orientation, anxiety, and 
perceiving mathematics to have low importance, do not appear to be associ- 
ated with students’ choices to game the system. 

On a broad scale, it appears that gaming is associated with negative affec- 
tive states. Students who game are likely to dislike the subject matter (H3), 
computers (H4), the learning environment (potentially) (H5), and to feel 
frustration (H12), and that the tutor is not helpful with learning (H8). There 
also appears to be some relationship between believing that effort is not 
helpful (H7) and is unenjoyable (H10). 

The evidence from these three studies agrees with some of the predictions 
made by our teachers, but strongly disagrees with other predictions. The 
teachers thought that gaming was caused by frustration (H12) and lack of 
educational self-drive (H10); the results from the studies agree. However, a 
third category that teachers thought was related to student gaming was per- 
formance goals (H1), which the study results do not support. In addition, the 
teachers did not expect either disliking the subject matter (H3) or disliking 
computers (H4) to be related to gaming the system, though both were, and 
other hypotheses such as the belief of mathematics as an innate ability (H7), 
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the idea of disliking the learning environment (HS5) and the idea that the tutor 
is not helpful for learning (H8), were strongly predicted against by the teach- 
ers. Hence, the teachers appear to have gotten some key predictions correct, 
but in many other ways their intuitions were not accurate: it is worth noting, 
however, that on hypothesis H1, at least two research groups published the 
same (incorrect) prediction as the teachers. 

Another interesting finding is that the characteristics associated with 
gaming are reasonably coherent between the Cognitive Tutors and ASSIST- 
ments. The two environments, while both based on cognitive models of stu- 
dent learning, have fairly different pedagogical characteristics and are used 
by different populations. In addition, the definitions of gaming behavior dif- 
fered between the two environments and research groups. Despite these dif- 
ferences, there were strong commonalities between the patterns of student 
characteristics associated with gaming behavior in each environment. This 
suggests that the results obtained here are reasonably likely to be general 
across learning environments, rather than specific to a single learning envi- 
ronment. Since gaming has been observed in a wide variety of environ- 
ments, it is possible that the results described here will also explain gaming 
behavior in learning environments beyond those described here. 

In the following section, we discuss how it may be possible to design sys- 
tems that respond not just to gaming when it occurs (Walonoski & Heffernan, 
2006b; Baker, Corbett, Koedinger, Evenson, et al., 2006) but to the root caus- 
es that underlie students’ decisions to game the system. Knowing that gam- 
ing the system is related to the presence of negative affective states opens the 
possibility that we can reduce gaming by improving students’ affect. 


DESIGN IMPLICATIONS 


Ultimately, the goal of the research discussed here is to inform the design 
of systems that will reduce or even eliminate student motivation to game the 
system, and, in turn, to improve students’ learning. The results we obtain 
have many design implications for educational software, and we propose 
some new methods and discuss some existing methods to address the prob- 
lems poised by our findings. 

One possible solution is to pay more attention to problem difficulty. Giv- 
ing items which are challenging but which students do not find too difficult 
is a delicate problem of balance (Vygotsky, 1978; Arroyo & Murray, 2002). 
But solving that problem offers the possibility of alleviating student frustra- 
tion related to problem difficulty. This simple suggestion is consistent both 
with our findings in this article, and with evidence that harmful gamers 
game on the hardest steps (Baker, Corbett, & Koedinger, 2004). Making 
problems easier may have other motivational benefits as well. However, if 
implemented without care, it may reduce learning (Paas & Van Merrienboer, 
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1994). Alternatively, it may be possible to reduce frustration by offering 
more direct, concise, and unambiguous hints and feedback messages. 

Providing shorter problems might also reduce frustration and dislike of 
the learning environment. Some problems within both systems are currently 
very long (six to nine steps with multi-level hints at each step, possibly with 
nested subproblems). If a problem is short, a student should be able to com- 
plete it in a few minutes. Completing many problems in a reasonably short 
time may increase students’ feelings of accomplishment, and improve their 
self-drive within the tutor context. Similarly, shorter lessons or problem sets 
which are achievable in a single class period may increase students’ feelings 
of accomplishment. 

Another option is to allow students to skip problem steps or even entire 
problems, in another attempt to curb frustration related to problem difficul- 
ty and lack of control. The tutoring system could simply return to the 
skipped problem later to ensure that all the tutoring content is covered. The 
ability to skip problems could be tied to problem difficulty, student knowl- 
edge, and/or the student’s rate of gaming. 

A related way to address frustration with problem difficulty and lack of 
control associated with the learning environment would be to allow students 
to select the next problem or to take turns with the tutor in selecting the next 
problem (Mostow et al., 2002; Mitrovic & Martin, 2003). The selection 
process could possibly involve the display of problem previews or thumb- 
nails, perhaps with associated difficulty information or subject identifica- 
tion. Simply by providing an element of choice, even among limited and 
similar options, may give students just enough control over the environment 
that they’re looking for. 

While there are many potential techniques for addressing frustration with 
the learning environment, some factors associated with gaming, such as dis- 
liking computers or mathematics, seem on first glance to be beyond the 
scope of what can be easily addressed within interactive learning environ- 
ment. It is hard to know how we can change those perceptions directly. 
However, by addressing the other problems, and having minimally-frustrat- 
ing tutoring systems with learning environments that students think are fun, 
perhaps it will become possible to develop environments that succeed even 
for students who dislike both computers and the subject matter. 


CONCLUSION 


In this article, we have presented four studies about what motivations, 
attitudes, and affective states are associated with the choice to game. Two 
studies investigated these issues within the context of Cognitive Tutors, cor- 
relating students’ self-reports with their gaming behavior as assessed by a 
machine-learned detector. Two other studies investigated these issues with- 
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in the ASSISTments system. One ASSISTments study correlated students’ 
self-reports with their gaming behavior as assessed by a rationally developed 
detector. The other ASSISTments study investigated teachers’ beliefs of why 
students game the system. 

Interestingly, despite the fairly different methods used in these studies, a 
reasonably consistent overall profile of gaming students can be seen across 
the studies. Across studies with two different systems, we found that dislik- 
ing the software’s subject matter (H3) and lack of self-drive (H10) are asso- 
ciated with a student’s choice to game. Other constructs such as frustration 
(H12), disliking computers (H4) and the learning environment (HS), believ- 
ing that mathematics ability is innate (H7), and believing that the tutor is not 
helpful for learning (H8), were associated with the choice to game in at least 
one of the three studies on student behavior and motivation. These con- 
structs may be worth investigating in future studies. 

Perhaps the biggest surprise from our studies regards the relationship 
between performance goals and gaming (H1). Performance goals were 
hypothesized both in prior research on gaming (Baker, Corbett, Koedinger, 
& Wagner, et al., 2004; Martinez-Miron et al., 2004) and by teachers expe- 
rienced with using the ASSISTments system in class, to be a major driving 
factor for why students game, but none of the three studies on student behav- 
ior and motivation found any evidence for a relationship between perfor- 
mance goals and gaming behavior. 

Overall, the teachers’ hypotheses about gaming were only partially cor- 
rect. The teachers correctly predicted that lack of self-drive (H10) and frus- 
tration (H12) would be associated with gaming, but did not predict the 
strong relationship between gaming and disliking the subject matter (H3). 
The teachers, like the researchers, predicted a relationship between gaming 
and performance goals that was not found (H1). Given the incomplete match 
between the teachers’ predictions and our results, the results of the three 
studies on student motivation presented here may be of use to teachers who 
use interactive learning environments in their classes, enabling them to 
respond to gaming more effectively. 

Understanding why students choose to game is useful in multiple ways. As 
discussed in the introduction, gaming the system occurs in many types of 
educational software, and is associated with significantly worse learning 
across several prior studies. Understanding why students choose to game sys- 
tems is a key step towards developing systems that students do not choose to 
game, and training teachers who use interactive learning environments in 
their classrooms, to improve learning outcomes for all students. In addition, 
these modified systems may be more appropriate and more effective for stu- 
dents who experience the same affective states as students who game, but 
who do not choose to game themselves. In addition, understanding why stu- 
dents game educational systems may give insight on other problems within 
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human-computer interaction. Abuse and misuse of interactive technology has 
become recognized as an increasing problem within human-computer inter- 
action as a field (De Angeli, Brahnam, Wallis, & Dix, 2006). It is possible that 
our findings about this type of software misuse may shed light on why other 
users choose to misuse other types of interactive technology. 
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Appendix 
One Version of the Teacher Survey 
Dear Teacher. We are investigating why some student’s might engage in “Gaming” behavior with the ASSIST- 


ment system. By gam 
, abusing the hints by clic 


for example 


ing every answer in multiple choice | 


students might game. 


This is a lis 


1, 2, or 3. Also, mark any reasons that you think are clearly not related to student gaming with an X. 


Student Goals 
ing performance goals instead of learning goals: Student think their goal should be to get through 


Hav! 
as n 
tod 


0. 





Des! 


ing we mean subverting the software to complete problems without having to learn: 


king through the hints to get the answer, or systematic guessing (try- 
uestions, or answering 1,2,3,4,5,6...). We want your opinion on why 


of possible reasons: Please mark your top three guesses below on why students game with a 











any items as possible as opposed to learning to answer the questions that they don’t know how 











re For More Control: ASSISTments do not provide the student with enough control over their 


earning, so they game to reassert more control. 


Student Attitudes 








Believing that students do wel 
of their effort 


Believ 





Believ 


ng the ASSISTment sys 











iking Mathematics: Student game because they do not like mathematics 
iking Computers in general: Student game because they do not like working with computers. 
iking the ASSISTment sys' 


Student Beliefs 
Believing mathematics is not 


em in specific. 


mportant 


or poorly at math because of their innate mathematical ability, instead 


em is Not Helpful For Learning 





ng the ASSISTment sys 


Student General Approaches 





no 


em is Uncaring 


Students game the system because they lack self-drive, especially for completing tasks which do 
personally motivate them. 


Students who are generally passive-aggressive game the system in order to resist a task they don’t 
like without appearing to be d 








Student Emotions 








loing so 


Frustration: Student game because they are frustrated. 


Students game because they feel Anxiety 





Remembe 


to mark any reasons that you think are clearly not the cause of student gaming with an X. 


Do you have any comments, or guesses or hypotheses that we missed? 


